这项工作提出了下一代人类机器人界面，只能通过视觉来推断和实现用户的操纵意图。具体而言，我们开发了一个集成了近眼跟踪和机器人操作的系统，以实现用户指定的操作（例如，抓取，拾取和位置等），在其中将视觉信息与人类的注意合并在一起，以创建为所需的映射机器人动作。为了实现视力指导的操纵，开发了一个头部安装的近眼跟踪设备，以实时跟踪眼球运动，以便可以确定用户的视觉注意力。为了提高抓地力性能，然后开发出基于变压器的GRASP模型。堆叠的变压器块用于提取层次特征，其中在每个阶段扩展了通道的体积，同时挤压了特征地图的分辨率。实验验证表明，眼球跟踪系统产生低的凝视估计误差，抓地力系统在多个握把数据集上产生有希望的结果。这项工作是基于凝视互动的辅助机器人的概念证明，该机器人具有巨大的希望，可以帮助老年人或上肢残疾在日常生活中。可在\ url {https://www.youtube.com/watch?v=yuz1hukyurm}上获得演示视频。

点云是3D内容的至关重要表示，在虚拟现实，混合现实，自动驾驶等许多领域已广泛使用，随着数据中点数的增加，如何有效地压缩点云变为一个具有挑战性的问题。在本文中，我们提出了一组基于贴片的点云压缩的重大改进，即用于熵编码的可学习上下文模型，用于采样质心点的OCTREE编码以及集成的压缩和训练过程。此外，我们提出了一个对抗网络，以改善重建过程中点的均匀性。我们的实验表明，改进的基于斑块的自动编码器在稀疏和大规模点云上的速率延伸性能方面优于最先进的。更重要的是，我们的方法可以在确保重建质量的同时保持短时间的压缩时间。

由于其在自主驾驶中的应用，因此基于单眼图像的3D感知已成为一个活跃的研究领域。与基于激光雷达的技术相比，单眼3D感知（包括检测和跟踪）的方法通常会产生较低的性能。通过系统的分析，我们确定了每个对象深度估计精度是界限性能的主要因素。在这种观察过程中，我们提出了一种多级融合方法，该方法将不同的表示（RGB和伪LIDAR）和跨多个对象（Tracklets）的时间信息结合在一起，以增强对目标深度估计。我们提出的融合方法实现了Waymo打开数据集，KITTI检测数据集和Kitti MOT数据集的每个对象深度估计的最新性能。我们进一步证明，通过简单地用融合增强的深度替换估计的深度，我们可以在单眼3D感知任务（包括检测和跟踪）方面取得重大改进。

New architecture GPUs like A100 are now equipped with multi-instance GPU (MIG) technology, which allows the GPU to be partitioned into multiple small, isolated instances. This technology provides more flexibility for users to support both deep learning training and inference workloads, but efficiently utilizing it can still be challenging. The vision of this paper is to provide a more comprehensive and practical benchmark study for MIG in order to eliminate the need for tedious manual benchmarking and tuning efforts. To achieve this vision, the paper presents MIGPerf, an open-source tool that streamlines the benchmark study for MIG. Using MIGPerf, the authors conduct a series of experiments, including deep learning training and inference characterization on MIG, GPU sharing characterization, and framework compatibility with MIG. The results of these experiments provide new insights and guidance for users to effectively employ MIG, and lay the foundation for further research on the orchestration of hybrid training and inference workloads on MIGs. The code and results are released on https://github.com/MLSysOps/MIGProfiler. This work is still in progress and more results will be published soon.

An unbiased scene graph generation (SGG) algorithm referred to as Skew Class-balanced Re-weighting (SCR) is proposed for considering the unbiased predicate prediction caused by the long-tailed distribution. The prior works focus mainly on alleviating the deteriorating performances of the minority predicate predictions, showing drastic dropping recall scores, i.e., losing the majority predicate performances. It has not yet correctly analyzed the trade-off between majority and minority predicate performances in the limited SGG datasets. In this paper, to alleviate the issue, the Skew Class-balanced Re-weighting (SCR) loss function is considered for the unbiased SGG models. Leveraged by the skewness of biased predicate predictions, the SCR estimates the target predicate weight coefficient and then re-weights more to the biased predicates for better trading-off between the majority predicates and the minority ones. Extensive experiments conducted on the standard Visual Genome dataset and Open Image V4 \& V6 show the performances and generality of the SCR with the traditional SGG models.

Learning to predict masked tokens in a sequence has been shown to be a powerful pretraining objective for large-scale language models. After training, such masked language models can provide distributions of tokens conditioned on bidirectional context. In this short draft, we show that such bidirectional conditionals often demonstrate considerable inconsistencies, i.e., they can not be derived from a coherent joint distribution when considered together. We empirically quantify such inconsistencies in the simple scenario of bigrams for two common styles of masked language models: T5-style and BERT-style. For example, we show that T5 models often confuse its own preference regarding two similar bigrams. Such inconsistencies may represent a theoretical pitfall for the research work on sampling sequences based on the bidirectional conditionals learned by BERT-style MLMs. This phenomenon also means that T5-style MLMs capable of infilling will generate discrepant results depending on how much masking is given, which may represent a particular trust issue.

This paper presents a practical global optimization algorithm for the K-center clustering problem, which aims to select K samples as the cluster centers to minimize the maximum within-cluster distance. This algorithm is based on a reduced-space branch and bound scheme and guarantees convergence to the global optimum in a finite number of steps by only branching on the regions of centers. To improve efficiency, we have designed a two-stage decomposable lower bound, the solution of which can be derived in a closed form. In addition, we also propose several acceleration techniques to narrow down the region of centers, including bounds tightening, sample reduction, and parallelization. Extensive studies on synthetic and real-world datasets have demonstrated that our algorithm can solve the K-center problems to global optimal within 4 hours for ten million samples in the serial mode and one billion samples in the parallel mode. Moreover, compared with the state-of-the-art heuristic methods, the global optimum obtained by our algorithm can averagely reduce the objective function by 25.8% on all the synthetic and real-world datasets.

Through a study of multi-gas mixture datasets, we show that in multi-component spectral analysis, the number of functional or non-functional principal components required to retain the essential information is the same as the number of independent constituents in the mixture set. Due to the mutual in-dependency among different gas molecules, near one-to-one projection from the principal component to the mixture constituent can be established, leading to a significant simplification of spectral quantification. Further, with the knowledge of the molar extinction coefficients of each constituent, a complete principal component set can be extracted from the coefficients directly, and few to none training samples are required for the learning model. Compared to other approaches, the proposed methods provide fast and accurate spectral quantification solutions with a small memory size needed.

Neural operators, which emerge as implicit solution operators of hidden governing equations, have recently become popular tools for learning responses of complex real-world physical systems. Nevertheless, the majority of neural operator applications has thus far been data-driven, which neglects the intrinsic preservation of fundamental physical laws in data. In this paper, we introduce a novel integral neural operator architecture, to learn physical models with fundamental conservation laws automatically guaranteed. In particular, by replacing the frame-dependent position information with its invariant counterpart in the kernel space, the proposed neural operator is by design translation- and rotation-invariant, and consequently abides by the conservation laws of linear and angular momentums. As applications, we demonstrate the expressivity and efficacy of our model in learning complex material behaviors from both synthetic and experimental datasets, and show that, by automatically satisfying these essential physical laws, our learned neural operator is not only generalizable in handling translated and rotated datasets, but also achieves state-of-the-art accuracy and efficiency as compared to baseline neural operator models.

In the field of cross-modal retrieval, single encoder models tend to perform better than dual encoder models, but they suffer from high latency and low throughput. In this paper, we present a dual encoder model called BagFormer that utilizes a cross modal interaction mechanism to improve recall performance without sacrificing latency and throughput. BagFormer achieves this through the use of bag-wise interactions, which allow for the transformation of text to a more appropriate granularity and the incorporation of entity knowledge into the model. Our experiments demonstrate that BagFormer is able to achieve results comparable to state-of-the-art single encoder models in cross-modal retrieval tasks, while also offering efficient training and inference with 20.72 times lower latency and 25.74 times higher throughput.